Enforce compute cluster version #1601
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dakinggg
reviewed
Oct 18, 2024
| @@ -451,6 +451,7 @@ def convert_dataset_hf_from_args( | |||
| ValueError: If the output directory already contains the requested splits | |||
| ValueError: If `concat_tokens` is set but `tokenizer` is not | |||
| """ | |||
| os.environ['WORLD_SIZE'] = '1' | |||
There was a problem hiding this comment.
are these changes supposed to be included?
There was a problem hiding this comment.
Actually going to close this pr, during testing seems like every version I tested worked (strangely), so perhaps old bug was actually transient, and was solved not by the version change but simply restarting the cluster. Will revisit if it comes up again.
There was a problem hiding this comment.
Original motivation is we don't know what works and what gets deprecated
added 2 commits
October 18, 2024 15:16
|
Not needed as all versions currently work... see above comment. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
To resolve:
https://databricks.slack.com/archives/C06RXC1NPQC/p1727371779924089
Essentially, we check if the compute runtime version is higher than our
databricks-connectversion to determine whether to use 'dbconnect' or not, but we don't check if the compute runtime version is too high. There is no formula to determine which version is too high, so we should just enforce the customer using a computer that has the same major runtime as thedatabricks-connectversionTest:
Error should be thrown:
with the following compute:
But should work: